kestrel网络编程--开发Fiddler
1 文章目的
本文讲解基于kestrel开发类似Fiddler应用的过程，让读者了解kestrel网络编程里面的kestrel中间件和http应用中间件。由于最终目的不是输出完整功能的产品，所以这里只实现Fiddler最核心的http请求和响应内容查看的功能。本文章是KestrelApp项目里面的一个demo的讲解，希望对您有用。
2 开发顺序
- 代理协议kestrel中间件
- tls协议侦测kestrel中间件
- 隧道和http协议侦测kestrel中间件
- 请求响应分析http中间件
- 反向代理http中间件
- 编排中间件创建服务器和应用
3 传输层与kestrel中间件
所谓传输层，其目的是为了让应用协议数据安全、可靠、快速等传输而存在的一种协议，其特征是把应用协议的报文做为自己的负载，常见的tcp、udp、quic、tls等都可以理解为传输层协议。
比如http协议，常见有如下的传输方式：
httpovertcphttpovertlsovertcphttpoverquicoverudp
3.1 Fiddler的传输层
Fiddler要处理以下三种http传输情况：
httpovertcp：直接http请求首页httpoverproxyovertcp：代理http流量httpovertlsoverproxyovertcp：代理https流量
3.2 Kestrel的中间件
kestrel目前的传输层基于tcp或quic两种，同时内置了tls中间件，需要调用ListenOptions.UseHttps()来使用tls中间件。kestrel的中间件的表现形式为：Func<ConnectionDelegate, ConnectionDelegate>，为了使用读者能够简单理解中间件，我在KestrelFramework里定义了kestrel中间件的变种接口，大家基于此接口来实现更多的中间件就方便很多：
/// <summary>
/// Kestrel的中间件接口
/// </summary>
public interface IKestrelMiddleware
{
/// <summary>
/// 执行
/// </summary>
/// <param name="next"></param>
/// <param name="context"></param>
/// <returns></returns>
Task InvokeAsync(ConnectionDelegate next, ConnectionContext context);
}
4 代理协议kestrel中间件
Filddler最基础的功能是它是一个http代理服务器， 我们需要为kestrel编写代理中间件，用于处理代理传输层。http代理协议分两种：普通的http代理和Connect隧道代理。两种的报文者是遵循http1.0或1.1的文本格式，我们可以使用kestrel自带的HttpParser<>来解析这些复杂的http文本协议。
4.1 代理特征
在中间件编程模式中，Feature是一个很重要的中间件沟通桥梁，它往往是某个中间件工作之后，留下的财产，让之后的中间件来获取并受益。我们的代理中间件，也设计了IProxyFeature，告诉之后的中间件一些代理特征。
/// <summary>
/// 代理Feature
/// </summary>
public interface IProxyFeature
{
/// <summary>
/// 代理主机
/// </summary>
HostString ProxyHost { get; }
/// <summary>
/// 代理协议
/// </summary>
ProxyProtocol ProxyProtocol { get; }
}
/// <summary>
/// 代理协议
/// </summary>
public enum ProxyProtocol
{
/// <summary>
/// 无代理
/// </summary>
None,
/// <summary>
/// http代理
/// </summary>
HttpProxy,
/// <summary>
/// 隧道代理
/// </summary>
TunnelProxy
}
4.2 代理中间件的实现
/// <summary>
/// 代理中间件
/// </summary>
sealed class KestrelProxyMiddleware : IKestrelMiddleware
{
private static readonly HttpParser<HttpRequestHandler> httpParser = new();
private static readonly byte[] http200 = Encoding.ASCII.GetBytes("HTTP/1.1 200 Connection Established\r\n\r\n");
private static readonly byte[] http400 = Encoding.ASCII.GetBytes("HTTP/1.1 400 Bad Request\r\n\r\n");
/// <summary>
/// 解析代理
/// </summary>
/// <param name="next"></param>
/// <param name="context"></param>
/// <returns></returns>
public async Task InvokeAsync(ConnectionDelegate next, ConnectionContext context)
{
var input = context.Transport.Input;
var output = context.Transport.Output;
var request = new HttpRequestHandler();
while (context.ConnectionClosed.IsCancellationRequested == false)
{
var result = await input.ReadAsync();
if (result.IsCanceled)
{
break;
}
try
{
if (ParseRequest(result, request, out var consumed))
{
if (request.ProxyProtocol == ProxyProtocol.TunnelProxy)
{
input.AdvanceTo(consumed);
await output.WriteAsync(http200);
}
else
{
input.AdvanceTo(result.Buffer.Start);
}
context.Features.Set<IProxyFeature>(request);
await next(context);
break;
}
else
{
input.AdvanceTo(result.Buffer.Start, result.Buffer.End);
}
if (result.IsCompleted)
{
break;
}
}
catch (Exception)
{
await output.WriteAsync(http400);
break;
}
}
}
/// <summary>
/// 解析http请求
/// </summary>
/// <param name="result"></param>
/// <param name="request"></param>
/// <param name="consumed"></param>
/// <returns></returns>
private static bool ParseRequest(ReadResult result, HttpRequestHandler request, out SequencePosition consumed)
{
var reader = new SequenceReader<byte>(result.Buffer);
if (httpParser.ParseRequestLine(request, ref reader) &&
httpParser.ParseHeaders(request, ref reader))
{
consumed = reader.Position;
return true;
}
else
{
consumed = default;
return false;
}
}
/// <summary>
/// 代理请求处理器
/// </summary>
private class HttpRequestHandler : IHttpRequestLineHandler, IHttpHeadersHandler, IProxyFeature
{
private HttpMethod method;
public HostString ProxyHost { get; private set; }
public ProxyProtocol ProxyProtocol
{
get
{
if (ProxyHost.HasValue == false)
{
return ProxyProtocol.None;
}
if (method == HttpMethod.Connect)
{
return ProxyProtocol.TunnelProxy;
}
return ProxyProtocol.HttpProxy;
}
}
void IHttpRequestLineHandler.OnStartLine(HttpVersionAndMethod versionAndMethod, TargetOffsetPathLength targetPath, Span<byte> startLine)
{
method = versionAndMethod.Method;
var host = Encoding.ASCII.GetString(startLine.Slice(targetPath.Offset, targetPath.Length));
if (versionAndMethod.Method == HttpMethod.Connect)
{
ProxyHost = HostString.FromUriComponent(host);
}
else if (Uri.TryCreate(host, UriKind.Absolute, out var uri))
{
ProxyHost = HostString.FromUriComponent(uri);
}
}
void IHttpHeadersHandler.OnHeader(ReadOnlySpan<byte> name, ReadOnlySpan<byte> value)
{
}
void IHttpHeadersHandler.OnHeadersComplete(bool endStream)
{
}
void IHttpHeadersHandler.OnStaticIndexedHeader(int index)
{
}
void IHttpHeadersHandler.OnStaticIndexedHeader(int index, ReadOnlySpan<byte> value)
{
}
}
}
5 tls协议侦测kestrel中间件
Fiddler只监听了一个端口，要同时支持非加密和加密两种流量，如果不调用调用ListenOptions.UseHttps()，我们的程序就不支持https的分析；如果直接调用ListenOptions.UseHttps()，会让我们的程序不支持非加密的http的分析，这就要求我们有条件的根据客户端发来的流量分析是否需要开启。
我已经在KestrelFramework内置了TlsDetection中间件，这个中间件可以根据客户端的实际流量类型来选择是否使用tls。在Fiddler中，我们还需要根据客户端的tls握手中的sni使用ca证书来动态生成服务器证书用于tls加密传输。
/// <summary>
/// 证书服务
/// </summary>
sealed class CertService
{
private const string CACERT_PATH = "cacert";
private readonly IMemoryCache serverCertCache;
private readonly IEnumerable<ICaCertInstaller> certInstallers;
private readonly ILogger<CertService> logger;
private X509Certificate2? caCert;
/// <summary>
/// 获取证书文件路径
/// </summary>
public string CaCerFilePath { get; } = OperatingSystem.IsLinux() ? $"{CACERT_PATH}/fiddler.crt" : $"{CACERT_PATH}/fiddler.cer";
/// <summary>
/// 获取私钥文件路径
/// </summary>
public string CaKeyFilePath { get; } = $"{CACERT_PATH}/fiddler.key";
/// <summary>
/// 证书服务
/// </summary>
/// <param name="serverCertCache"></param>
/// <param name="certInstallers"></param>
/// <param name="logger"></param>
public CertService(
IMemoryCache serverCertCache,
IEnumerable<ICaCertInstaller> certInstallers,
ILogger<CertService> logger)
{
this.serverCertCache = serverCertCache;
this.certInstallers = certInstallers;
this.logger = logger;
Directory.CreateDirectory(CACERT_PATH);
}
/// <summary>
/// 生成CA证书
/// </summary>
public bool CreateCaCertIfNotExists()
{
if (File.Exists(this.CaCerFilePath) && File.Exists(this.CaKeyFilePath))
{
return false;
}
File.Delete(this.CaCerFilePath);
File.Delete(this.CaKeyFilePath);
var notBefore = DateTimeOffset.Now.AddDays(-1);
var notAfter = DateTimeOffset.Now.AddYears(10);
var subjectName = new X500DistinguishedName($"CN={nameof(Fiddler)}");
this.caCert = CertGenerator.CreateCACertificate(subjectName, notBefore, notAfter);
var privateKeyPem = this.caCert.GetRSAPrivateKey()?.ExportRSAPrivateKeyPem();
File.WriteAllText(this.CaKeyFilePath, new string(privateKeyPem), Encoding.ASCII);
var certPem = this.caCert.ExportCertificatePem();
File.WriteAllText(this.CaCerFilePath, new string(certPem), Encoding.ASCII);
return true;
}
/// <summary>
/// 安装和信任CA证书
/// </summary>
public void InstallAndTrustCaCert()
{
var installer = this.certInstallers.FirstOrDefault(item => item.IsSupported());
if (installer != null)
{
installer.Install(this.CaCerFilePath);
}
else
{
this.logger.LogWarning($"请根据你的系统平台手动安装和信任CA证书{this.CaCerFilePath}");
}
}
/// <summary>
/// 获取颁发给指定域名的证书
/// </summary>
/// <param name="domain"></param>
/// <returns></returns>
public X509Certificate2 GetOrCreateServerCert(string? domain)
{
if (this.caCert == null)
{
using var rsa = RSA.Create();
rsa.ImportFromPem(File.ReadAllText(this.CaKeyFilePath));
this.caCert = new X509Certificate2(this.CaCerFilePath).CopyWithPrivateKey(rsa);
}
var key = $"{nameof(CertService)}:{domain}";
var endCert = this.serverCertCache.GetOrCreate(key, GetOrCreateCert);
return endCert!;
// 生成域名的1年证书
X509Certificate2 GetOrCreateCert(ICacheEntry entry)
{
var notBefore = DateTimeOffset.Now.AddDays(-1);
var notAfter = DateTimeOffset.Now.AddYears(1);
entry.SetAbsoluteExpiration(notAfter);
var extraDomains = GetExtraDomains();
var subjectName = new X500DistinguishedName($"CN={domain}");
var endCert = CertGenerator.CreateEndCertificate(this.caCert, subjectName, extraDomains, notBefore, notAfter);
// 重新初始化证书，以兼容win平台不能使用内存证书
return new X509Certificate2(endCert.Export(X509ContentType.Pfx));
}
}
/// <summary>
/// 获取域名
/// </summary>
/// <param name="domain"></param>
/// <returns></returns>
private static IEnumerable<string> GetExtraDomains()
{
yield return Environment.MachineName;
yield return IPAddress.Loopback.ToString();
yield return IPAddress.IPv6Loopback.ToString();
}
}
6 隧道和http协议侦测kestrel中间件
经过KestrelProxyMiddleware后的流量，在tls解密(如果可能)之后，一般情况下都是http流量了，但如果你在qq设置代理到我们这个伪Fildder之后，会发现部分流量流量不是http流量，原因是http隧道也是一个通用传输层，可以传输任意tcp或tcp之上的流量。所以我们需要新的中间件来检测当前流量，如果不是http流量就回退到隧道代理的流程，即我们不跟踪不分析这部分非http流量。
6.1 http流量侦测
/// <summary>
/// 流量侦测器
/// </summary>
private static class FlowDetector
{
private static readonly byte[] crlf = Encoding.ASCII.GetBytes("\r\n");
private static readonly byte[] http10 = Encoding.ASCII.GetBytes(" HTTP/1.0");
private static readonly byte[] http11 = Encoding.ASCII.GetBytes(" HTTP/1.1");
private static readonly byte[] http20 = Encoding.ASCII.GetBytes(" HTTP/2.0");
/// <summary>
/// 传输内容是否为http
/// </summary>
/// <param name="context"></param>
/// <returns></returns>
public static async ValueTask<bool> IsHttpAsync(ConnectionContext context)
{
var input = context.Transport.Input;
var result = await input.ReadAtLeastAsync(1);
var isHttp = IsHttp(result);
input.AdvanceTo(result.Buffer.Start);
return isHttp;
}
private static bool IsHttp(ReadResult result)
{
var reader = new SequenceReader<byte>(result.Buffer);
if (reader.TryReadToAny(out ReadOnlySpan<byte> line, crlf))
{
return line.EndsWith(http11) || line.EndsWith(http20) || line.EndsWith(http10);
}
return false;
}
}
6.2 隧道回退中间件
/// <summary>
/// 隧道传输中间件
/// </summary>
sealed class KestrelTunnelMiddleware : IKestrelMiddleware
{
private readonly ILogger<KestrelTunnelMiddleware> logger;
/// <summary>
/// 隧道传输中间件
/// </summary>
/// <param name="logger"></param>
public KestrelTunnelMiddleware(ILogger<KestrelTunnelMiddleware> logger)
{
this.logger = logger;
}
/// <summary>
/// 执行中间你件
/// </summary>
/// <param name="next"></param>
/// <param name="context"></param>
/// <returns></returns>
public async Task InvokeAsync(ConnectionDelegate next, ConnectionContext context)
{
var feature = context.Features.Get<IProxyFeature>();
if (feature == null || feature.ProxyProtocol == ProxyProtocol.None)
{
this.logger.LogInformation($"侦测到http直接请求");
await next(context);
}
else if (feature.ProxyProtocol == ProxyProtocol.HttpProxy)
{
this.logger.LogInformation($"侦测到普通http代理流量");
await next(context);
}
else if (await FlowDetector.IsHttpAsync(context))
{
this.logger.LogInformation($"侦测到隧道传输http流量");
await next(context);
}
else
{
this.logger.LogInformation($"跳过隧道传输非http流量{feature.ProxyHost}的拦截");
await TunnelAsync(context, feature);
}
}
/// <summary>
/// 隧道传输其它协议的数据
/// </summary>
/// <param name="context"></param>
/// <param name="feature"></param>
/// <returns></returns>
private async ValueTask TunnelAsync(ConnectionContext context, IProxyFeature feature)
{
var port = feature.ProxyHost.Port;
if (port == null)
{
return;
}
try
{
var host = feature.ProxyHost.Host;
using var socket = new Socket(SocketType.Stream, ProtocolType.Tcp);
await socket.ConnectAsync(host, port.Value, context.ConnectionClosed);
Stream stream = new NetworkStream(socket, ownsSocket: false);
// 如果有tls中间件，则反回来加密隧道
if (context.Features.Get<ITlsConnectionFeature>() != null)
{
var sslStream = new SslStream(stream, leaveInnerStreamOpen: true);
await sslStream.AuthenticateAsClientAsync(feature.ProxyHost.Host);
stream = sslStream;
}
var task1 = stream.CopyToAsync(context.Transport.Output);
var task2 = context.Transport.Input.CopyToAsync(stream);
await Task.WhenAny(task1, task2);
}
catch (Exception ex)
{
this.logger.LogError(ex, $"连接到{feature.ProxyHost}异常");
}
}
}
7 请求响应分析http中间件
这部分属于asp.netcore应用层内容，关键点是制作可多次读取的http请求body流和http响应body流，因为每个分析器实例都可以会重头读取一次请求内容和响应内容。
7.1 http分析器
为了方便各种分析器的独立实现，我们定义http分析器的接口
/// <summary>
/// http分析器
/// 支持多个实例
/// </summary>
public interface IHttpAnalyzer
{
/// <summary>
/// 分析http
/// </summary>
/// <param name="context"></param>
/// <returns></returns>
ValueTask AnalyzeAsync(HttpContext context);
}
这是输到日志的http分析器
public class LoggingHttpAnalyzer : IHttpAnalyzer
{
private readonly ILogger<LoggingHttpAnalyzer> logger;
public LoggingHttpAnalyzer(ILogger<LoggingHttpAnalyzer> logger)
{
this.logger = logger;
}
public async ValueTask AnalyzeAsync(HttpContext context)
{
var builder = new StringBuilder();
var writer = new StringWriter(builder);
writer.WriteLine("[REQUEST]");
await context.SerializeRequestAsync(writer);
writer.WriteLine("[RESPONSE]");
await context.SerializeResponseAsync(writer);
this.logger.LogInformation(builder.ToString());
}
}
7.2 分析http中间件
我们把请求body流和响应body流保存到临时文件，在所有分析器工作之后再删除。
/// <summary>
/// http分析中间件
/// </summary>
sealed class HttpAnalyzeMiddleware
{
private readonly RequestDelegate next;
private readonly IEnumerable<IHttpAnalyzer> analyzers;
/// <summary>
/// http分析中间件
/// </summary>
/// <param name="next"></param>
/// <param name="analyzers"></param>
public HttpAnalyzeMiddleware(
RequestDelegate next,
IEnumerable<IHttpAnalyzer> analyzers)
{
this.next = next;
this.analyzers = analyzers;
}
/// <summary>
/// 分析代理的http流量
/// </summary>
/// <param name="context"></param>
/// <returns></returns>
public async Task InvokeAsync(HttpContext context)
{
var feature = context.Features.Get<IProxyFeature>();
if (feature == null || feature.ProxyProtocol == ProxyProtocol.None)
{
await next(context);
return;
}
context.Request.EnableBuffering();
var oldBody = context.Response.Body;
using var response = new FileResponse();
try
{
// 替换response的body
context.Response.Body = response.Body;
// 请求下个中间件
await next(context);
// 处理分析
await this.AnalyzeAsync(context);
}
finally
{
response.Body.Position = 0L;
await response.Body.CopyToAsync(oldBody);
context.Response.Body = oldBody;
}
}
private async ValueTask AnalyzeAsync(HttpContext context)
{
foreach (var item in this.analyzers)
{
context.Request.Body.Position = 0L;
context.Response.Body.Position = 0L;
await item.AnalyzeAsync(context);
}
}
private class FileResponse : IDisposable
{
private readonly string filePath = Path.GetTempFileName();
public Stream Body { get; }
public FileResponse()
{
this.Body = new FileStream(filePath, FileMode.Open, FileAccess.ReadWrite);
}
public void Dispose()
{
this.Body.Dispose();
File.Delete(filePath);
}
}
}
8 反向代理http中间件
我们需要把请求转发到真实的目标服务器，这时我们的应用程序是一个http客户端角色，这个过程与nginx的反向代理是一致的。具体的实现上，我们直接使用yarp库来完成即可。
/// <summary>
/// http代理执行中间件
/// </summary>
sealed class HttpForwardMiddleware
{
private readonly RequestDelegate next;
private readonly IHttpForwarder httpForwarder;
private readonly HttpMessageInvoker httpClient = new(CreateSocketsHttpHandler());
/// <summary>
/// http代理执行中间件
/// </summary>
/// <param name="next"></param>
/// <param name="httpForwarder"></param>
public HttpForwardMiddleware(
RequestDelegate next,
IHttpForwarder httpForwarder)
{
this.next = next;
this.httpForwarder = httpForwarder;
}
/// <summary>
/// 转发http流量
/// </summary>
/// <param name="context"></param>
/// <returns></returns>
public async Task InvokeAsync(HttpContext context)
{
var feature = context.Features.Get<IProxyFeature>();
if (feature == null || feature.ProxyProtocol == ProxyProtocol.None)
{
await next(context);
}
else
{
var scheme = context.Request.Scheme;
var destinationPrefix = $"{scheme}://{feature.ProxyHost}";
await httpForwarder.SendAsync(context, destinationPrefix, httpClient, ForwarderRequestConfig.Empty, HttpTransformer.Empty);
}
}
private static SocketsHttpHandler CreateSocketsHttpHandler()
{
return new SocketsHttpHandler
{
Proxy = null,
UseProxy = false,
UseCookies = false,
AllowAutoRedirect = false,
AutomaticDecompression = DecompressionMethods.None,
};
}
}
9 编排中间件创建服务器和应用
9.1 kestrel中间件编排
这里要特别注意顺序，传输层套娃。
/// <summary>
/// ListenOptions扩展
/// </summary>
public static partial class ListenOptionsExtensions
{
/// <summary>
/// 使用Fiddler的kestrel中间件
/// </summary>
/// <param name="listen"></param>
public static ListenOptions UseFiddler(this ListenOptions listen)
{
// 代理协议中间件
listen.Use<KestrelProxyMiddleware>();
// tls侦测中间件
listen.UseTlsDetection(tls =>
{
var certService = listen.ApplicationServices.GetRequiredService<CertService>();
certService.CreateCaCertIfNotExists();
certService.InstallAndTrustCaCert();
tls.ServerCertificateSelector = (context, domain) => certService.GetOrCreateServerCert(domain);
});
// 隧道代理处理中间件
listen.Use<KestrelTunnelMiddleware>();
return listen;
}
}
9.2 http中间件的编排
public static class ApplicationBuilderExtensions
{
/// <summary>
/// 使用Fiddler的http中间件
/// </summary>
/// <param name="app"></param>
public static void UseFiddler(this IApplicationBuilder app)
{
app.UseMiddleware<HttpAnalyzeMiddleware>();
app.UseMiddleware<HttpForwardMiddleware>();
}
}
9.3 创建应用
我们可以在传统的MVC里创建伪fiddler的首页、下载证书等http交互页面。
public static void Main(string[] args)
{
var builder = WebApplication.CreateBuilder(args);
builder.Services
.AddFiddler()
.AddControllers();
builder.WebHost.ConfigureKestrel((context, kestrel) =>
{
var section = context.Configuration.GetSection("Kestrel");
kestrel.Configure(section).Endpoint("Fiddler", endpoint => endpoint.ListenOptions.UseFiddler());
});
var app = builder.Build();
app.UseRouting();
app.UseFiddler();
app.MapControllerRoute(
name: "default",
pattern: "{controller=Home}/{action=Index}/{id?}");
app.Run();
}
10 留给读者
如果让您来开发个伪Fiddler，除了本文的方法，您会使用什么方式来开发呢？
